Using Frame Semantics for Knowledge Extraction from Twitter

Knowledge bases have the potential to advance artificial intelligence, but often suffer from recall problems, i.e., lack of knowledge of new entities and relations. On the contrary, social media such as Twitter provide abundance of data, in a timely manner: information spreads at an incredible pace and is posted long before it makes it into more commonly used resources for knowledge extraction. In this paper we address the question whether we can exploit social media to extract new facts, which may at first seem like finding needles in haystacks. We collect tweets about 60 entities in Freebase and compare four methods to extract binary relation candidates, based on syntactic and semantic parsing and simple mechanism for factuality scoring. The extracted facts are manually evaluated in terms of their correctness and relevance for search. We show that moving from bottom-up syntactic or semantic dependency parsing formalisms to top-down framesemantic processing improves the robustness of knowledge extraction, producing more intelligible fact candidates of better quality. In order to evaluate the quality of frame semantic parsing on Twitter intrinsically, we make a multiply frame-annotated dataset of tweets publicly available. Knowledge extraction has primarily focused on mining Wikipedia and newswire data. For this reason, knowledge bases used for search such as Freebase suffer from low recall, only covering certain entity types, and only certain facts about those entities. Freebase, for example, contains the fact that the Walt Disney Company is a production company, but not that it, for instance, owns Marvel. A common problem in knowledge extraction is what is known as the reporting bias (Gordon and van Durme 2013), i.e., the fact that a lot of common knowledge is never made explicit. Social media platforms like Twitter have the potential to fill some of that gap, since they offer very different facts than what can be found in Wikipedia. People may tweet an obvious fact to inform their friends what they just realized, as a means of sarcasm, or simply to kill time. Finally, Twitter is a platform that potentially allows us to harvest facts in almost real time. E.g., a company may buy Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. up another company and tweet about it, long before the fact makes it into Wikipedia or Freebase. On the other hand, extracting useful facts from Twitter is a hard problem. Tweets often contain opinionated non-factual text, automated posts from third-party websites, and/or temporary facts that are irrelevant to search. True facts seem like needles in haystacks, but, on the other hand, the haystacks are plentiful on Twitter. There is also another reason that knowledge extraction from Twitter is hard. Most approaches to knowledge extraction rely on syntactico-semantic processing, and stateof-the-art parsing models fair badly on Twitter data (Foster et al. 2011), to the extent that it is prohibitive for downstream applications such as knowledge extraction. In this paper, we show that top-down frame semantic parsing is more robust to the domain shift from newswire to Twitter than other syntactico-semantic formalisms, and that this leads to more robust knowledge extraction. In particular, while syntactic and semantic dependency parsing models induced from newswire exhibit dramatic drops when applied to Twitter data, frame semantic parsing models seem to perform almost the same across domains. Our Approach We select 60 entities in Freebase distributed equally across persons, locations and organizations (see Table 1), and extract 70k tweets mentioning at least one of these entities. The data was collected during the summer 2014. We part of speech (POS) tag these tweets and pass the augmented tweets on to four different extraction models: a syntactic dependency parser, a semantic role labeler, a frame semantic parser, and a rule-based off-the-shelf (REVERB) open information extraction system (Fader, Soderland, and Etzioni 2011). For all systems, except REVERB, we apply the same heuristics to filter out relevant facts and rank them in terms of factuality using sentiment analysis. We evaluate facts in terms of their wellformedness, their correctness, and their relevance. We also ask subjects to rate triples in terms of opinionatedness for the sake of error analysis. Finally, we check the extracted facts for novelty against Freebase. Frame Semantic Parsing Frame semantic parsing is the task of assigning frames (Fillmore 1982) to text. Frames combine word sense disambiguation and semantic role labeling. The go-to ressource Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence